Heterogeneity in the usability evaluation process

Current prediction models for usability evaluations are based on stochastic distributions derived from series of Bernoulli processes. The underlying assumption of these models is a homogeneous detection probability despite of it being intuitively unrealistic. This paper contributes a simple statistical test for existence of heterogeneity in the process. The compound beta-binomial model is proposed to incorporate sources of heterogeneity and compared to the binomial model. Analysis of several data sets from the literature illustrates the methods and reveals that heterogeneity occurs in most situations. Finally, it is demonstrated how heterogeneity biases the prediction of evaluation processes. Open research questions are discussed and preliminary advice for practitioners for controlling their processes is given.